Extractor API #16
Extractor API #16
Conversation
psrpinto
force-pushed
the
extractor-api
branch
3 times, most recently
from
1bfda77 to
57cc696
Compare
akirk
force-pushed
the
simplify-directory-structure
branch
from
486ac09 to
ebd0919
Compare
psrpinto
force-pushed
the
extractor-api
branch
from
57cc696 to
070b712
Compare
Simplify directory structure
psrpinto
force-pushed
the
extractor-api
branch
from
b1f5e62 to
0a331b7
Compare
There was a problem hiding this comment.
What should happen if the extractor thinks it can handle the source but then can't?
What about having extractors report a confidence between 0 and 100 that they can extract the source?
Should an extractor extract a single type of data?
I think it should not be limited to a single type of data since parsing the DOM might give multiple pieces of data as side-effects. For example, followers and follows might be on the same page and thus could be extracted at the same time.
What concerns me more is how we'll register extractors considering that we might arrive at a large number of them. Maybe we can have a two stage system where we'd only add a subset of extractors based on the URL matching. WordPress for example could be added for all URLs except those that specifically match other matchers.
That way we wouldn't be registering all extractors on every page (since the content script loads into every page).
What about multi-language sources? We don't need to support it right away, but we should keep it in mind while designing the API.
👍
| * Source of data to be extracted, like a DOM document, a URL or any other kind of resource. | ||
| * For the moment, only DOM Document is supported. |
There was a problem hiding this comment.
Could you elaborate on these other kinds of resources? The URL can be accessed through document.location.href.
There was a problem hiding this comment.
The idea behind having a Source interface is so that the API does not depend on a specific data structure, that might not be available in all runtimes. If in the future we would like to run an extractor in nodejs, for example, the document would not exist (or a least would not have the same type).
Another reason would be that we can envision having extractors that don't rely on a document, but instead, for example, pull directly from a URL. (We could also make it so that an extractor can support multiple types of Sources, e.g. DOMSource and URLSource).
If we would not introduce the notion of a Source at this moment, adding support later for multiple types of sources would be a breaking change to the API, which would require updating all existing extractors.
This is something I hadn't considered. I think probably we should only run the content scripts if the extension is currently open. |
|
I will open a new PR to implement this when required. |
WIP
Doubts
Naming
Need better names for:
SiteDataSiteInfo?What should happen if the extractor thinks it can handle the source but then can't?
I guess we'd tell the user something like "Extractor didn't find anything, try another one"? Or, instead of having the extractor say whether it supports a source, should we ask the user right away which extractor they want to use?
Should an extractor extract a single type of data?
For example, should the
wordpress-restextractor extract posts and pages, or should there be awordpress-post-restand awordpress-page-restextractor?Having different extractors for different kinds of data would unlock the possibility of having specific extractors for certain data types. For example, there could be
wordpress-product-restwhat extracts products from an eccommerce site.What about multi-language sources?
We don't need to support it right away, but we should keep it in mind while designing the API.